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ABSTRACT 

Comprehensive examinations, long a bastion in many 
doctoral programs, are one of many customs under scrutiny for 
possible change in a movement towards more authentic means of 
educational assessment. This preliminary study surveyed chairs of 
departments of educational administration from universities across 
the United States to learn how computers and models of alternative 
assessment are changing the face of comprehensive examinations. 
Respondents were asked how often doctoral comprehensive examinations 
were administered, and asked to rate seven purposes derived from the 
literature on importance for administering comprehensive 
examinat i oni : accountability; tradition; gatekeeping; recall; rigor; 
application; and synthesis. They were asked whether or not they used 
each of Tour examination modes: handwriting; typing on computer; oral 
examination; and alternative assessments such as portfolios or 
pet f ii! mences . Questions also concerned the length of time students 
hart In complete examinations, and rating for each examination mode on 
a vaiietv »f iianea: student anxiety; public acceptance; rigor; 
i*lr '.'n,, r in pingtsm; relevance to future of candidate; faculty work 
Infill; am) potential for dishonesty. Ninety-one percent of respondents 
at know! edged that their departments allow students to use a computer 
an an alternative to handwriting the examination. Many also saw a 
need I in mm* authentic model of assessment as part of the doctoral 
pi man*, although f aw departments allowed forms other than written or 
otal tut Mi* roMpi titans i ve examination. Issues such as academic 
dishonesty in a computer eg**, the need for proctoring, and the mix of 
iHeiemra and other materials that should be allowed during the 
examination i^qmie further study. Two tables and one figure 
i I hut tat * data, (luntaini ill references.) (MAS) 
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Abstract 

The ever increasing role of computers in our society, together with a movement towards more 
authentic means of educational assessment, is impacting educators at all levels. Comprehensive 
examinations, long a bastion of tradition in many doctoral programs, are one of many customs 
under scrutiny for possible change. This preliminary study surv eyed Chairs in departments of 
Educational Administration from universities across the United States to learn how computers and 
models of alternative assessment are changing the face of comprehensiv e examinations. Ninety- 
one percent of the respondents acknowledged that their departments allow students to use a 
computer as an alternative to handwriting the examination Many also saw a need for more 
authentic modes of assessment as part of the doctoral process, although few departments allowed 
forms other than written or oral for the comprehensive examination Issues such as academic 
dishonesty in a computer age, the need for proctoring, and the mix of references and other 
materials that should be allowed during the examination require further study. 
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Using Computers to Write Comprehensive Examinations: 
A Study of Doctoral Level Examinations in Educational Administration Departments 

The comprehensive examination has been an essential component to scholars' work in 
universities for centuries. In the 1200s the Scholastic Model dominated, in which students gave 
oral demonstrations of their ability to synthesize learned materials. Not until 1871 was this 
process changed to include w ritten examinations. This new process became known as the 
Harvard Model. The innovation of the Harvard Model sought to "regulate and standardize 
requirements for graduation" (Manus, 1992, p. 678). Today, more than 100 years after the 
introduction of the Harvard Model, the methods of administration and purposes of comprehensive 
examinations remain essentially unchanged. 

Purposes for Comprehensive Examinations 

Purposes for administering comprehensive examinations, while more specifically 
articulated today than in the past, remain close to the purposes stated centuries ago. The synthesis 
required of students evaluated under the Scholastic Model in the thirteenth century can be likened 
to today's faculties use of comprehensive examinations to provide useful experiences for students. 
Students are provided an opportunity to organize and integrate what they have learned 
(Anderson, Krauskopf, Rogers, & NeaL 1984). Many also believe that the comprehensive 
examination is itself a learning experience for the student (Peterson, Bowman, Myer, & MaidI, 
1992). 

Other purposes that have been articulated relate more logically to the search for 
standardization that was reflected in the early Harvard Model Faculty may seek to standardize 
and test aspects of the program, such as its rigor (Burck & Peterson, 1983), the quality of its 
teachers (Peterson et al , 1992), or the quality of the overall program (Khanna & Khanna, 1972). 
Comprehensive examinations arc used to standardize outcomes by screening students for minimal 
skills (Anderson et al , 1984, Burck & Peteison, I'M, Khanna & Khanna, 1972; Manus, 1992; 
Murray, 1973. Pet ct son et al . 1992. Wolensky, 1979) They arc also seen as part of the tradition 
of higher education, in part a tile of passage tut the aspiring student (Anderson et al., 1984). 

Despite these seemingly clcai pui poses, dissatisfaction with the current form of 
comprehensive examinations is evident Some educators feel that the comprehensive examination 
is outdated and should be changed or eliminated (Khanna & Khanna, 1972) Others advocate 
alternative assessment models (Herman, Aschbacher, & Winters, 1992) The interest in alternative 
assessment strategies that has swept education at all levels may be seen as a reaction to the 
perception that many examination strategies call only for recall of fact, rather than for ability to 
use learned information (I lei man et al , 1992) Relevance and ability to use information has 
formed the crux of the debate regarding purposes for comprehensive examination. A similar 
focus is possible tegaiding the mode of comprehensive exam administration. Although current 
literature regauling methods of administering comprehensive examinations is scant, as recently as 
10 years ago, it was cleat that the unttcn examination given over several days in segments of 
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several hours was the most common means of administration (Anderson et al., 1984; Burck & 
Peterson, 1983). Oral examinations rarely substitute for, but often supplement, the written 
examination. 

Issues 

The current easy availability of computers, as well as the perception that knowledge of 
computers is relevant in today's world, has led to some changes in how students take 
examinations of all kinds, comprehensive examinations included (Gwinn & Beal, 1988; Hicken, 
1993; Wise & Plake, 1989; Wise & Plake, 1990). Computers are now used interactively 
(Wedman & Stefanich, 19S4), to generate examinations (Gwinn & Beal, 1988) and to type 
responses to essay questions (Peterson et al , 1992). With the proliferation of computers into all 
facets of education it is not surprising that comprehensiv e examinations are also impacted. 

If the purpose of administering comprehensive examinations remains to test factual recall, 
test security is an important issue. The examination instrument must remain hidden until its 
administration, an issue that has remained constant throughout the course of administering 
comprehensive examinations (Gwinn & Beal, 1988). However, the use of computers brings about 
another security issue. It is possible to "hide" information, either on a floppy diskette or on a 
computer's hard drive If the purpose of the exam is to test the student's natural recall, this 
"information hiding" must be prevented. Various measures have been taken to ensure students 
rely only on what is in their heads, including providing disks (Hicken, 1993). providing pre- 
cleaned computers, and close proctoring. 

Another issue concerns access to additional information Rather than concerning 
themselves with whether or not students had access to information only from their heads, some 
schools allow students to take portions of the examination on their own time, in their own space, 
using their own computers (Burck & Peterson, 1983; Peterson et al , 1992) No security measures 
are attempted This practice implies that what is important is the students 1 abilities to find, use, 
and make sense of information Like the Scholars Model of the thirteenth century, there is an 
implication that synthesis, as well as application arc important However, this model conflicts 
with the standatdi/alion sought by the llarvatd Model If students ate no longer tested in the 
same enviionment under the same strietuics, standardization can no longer be assumed. 

While thrse issues ii^aulmg the nattm of the examination atVcct examiners and students 
alike othn issiirs have a gti'itlci impait mi students I ho issue ol test anxiety, a constant in 
i omph hrnsiv r inanimation fulmimstiaiion (khamu iV Khanua l l *7?l. has even wider meaning 
with tin' usr nt » nmptitris AnxietN is no lonyri kisrd soh Is upon i|iu>siions of one's own ability, 
hut is rtlsn hasnl nn one's tinst in the im)i liptiii'iit In inu used hrspile the fact that most graduate 
MUthnts i ist' umiputeis constantly tlmniuhnut theii uuii^ woik students fcaicd computer 
nutlllmt lion rspri mllv when taking the examination within i oiistunned tune (Hicken, 1993). 
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It is clear that the Scholastic Model of providing an opportunity to synthesize information 
is still predominant in higher education today. However, other prerogatives are also coming to 
the fore. There is increasing concern that comprehensive examinations provide not only the 
opportunity to synthesize information, but also ensure that students are able to use information in 
real world settings This interest has led to some changes in comprehensive examination 
administration, including the incorporation of computers into the examination mode, and the 
beginnirig reliance on alternative forms of assessment, such as portfolios or presentations. 
Tensions arise, though, when the concerns for standardization evident in the Harvard Model meet 
the concern for ability to apply information 

Increased interest and heightened tension on the part of higher education faculties call for 
a revisiting of the purposes, rationales, and methods used to administer comprehensive 
examination. In this study, we report the results of a nation-wide survey of doctoral programs in 
educational administration. Although the small response rate (rp39) and the narrow scope of 
programs surveyed make this study preliminary, our findings suggest that indeed, faculties are 
looking for ways to change their comprehensive examination procedures 

Methods 

Respondents 

Respondents were chosen from the Educational Administration Directory , 1 1th Edition. 
In order to be included in the sample, the Educational Administration department had to offer a 
doctorate, either an Education Doctorate or a Doctor of Philosophy. Respondents were the 
department heads or chairs of these departments. 

Instrument 

A survey was constructed to discover various procedures, methods, attitudes and 
rationales for administering comprehensive examinations. Respondents were asked how often 
doctoral comprehensive examinations were administered. They were asked to rate seven 
pin poses derived I mm the literature on importance for administering comprehensive 
examinations They were asked whether or not they used each of four examination modes: 
handwriting, typing on computer, oral examination, and alternative assessments such as portfolios 
oi peifoimances I hev wore asked how long students had to complete examinations, and finally, 
to late each examination mode on a variety of issues 

The survey was administeied by telephone during a two week period. Answers were 
toiled on a sheet seannable by an optical maik reader Scanbook software was used to create a 
computer template which would then determine the format for the comma delimited ASCII file. 
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Collected surveys were scanned using the Scantron Optical Mark Reader, and the ensuing ASCII 
file was analyzed using SPSS for Windows. 

Procedure and Administration 

A total of 39 different department heads were interviewed. Results, summarized in Table 
2 below, show that most departments allowed both handwriting or the use of computers as the 
mode of administration Most departments proctored examinations, and did not allow outside 
references Examinations were most often given over the course of several days, encompassing 
several hours The mean number of administrations reflects the fact that some schools offered the 
examination once a year, while others offered it twice or three times a year. As noted, ten schools 
offered the examination on an individualized basis 
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•1 Vpuitmenls olVciing Altcniati\c Assessment models lo? completing compichcnsive examinations allowed anywhere 
Horn 2 months to the entire length of the doctoral progiam to complete the examination, especially when the examination 
consisted of ct eating a port folio. 

**l Acliuled ale the ten schools administering examinations on an individual basis 

t| Vpaitmcnts otVenng Oral examinations usually allowed notes, but no texts oi other references 

Purposes for administering comprehensive examinations have remained remarkably 
consistent throughout the course of their use However, some new purposes have arisen, perhaps 
in response to a demand that professional departments provide more real world experiences to 
their students While synthesis, with a mean score of 9 49 (on a scale from I to 10, 10 being the 
most important), remained the most important reason reported for administering comprehensive 
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examinations, application was almost as important, with a mean score of 8,41. Ensuring rigor 
was also important with a mean score of 7. 15. The other four purposes, accountability, 
maintaining tradition, gatekeeping, and testing recall, had similar, and lower, means (M = 4.15, 
4.31, 4.33, 4.62, respectively). Figure 1 summarizes these findings. 
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Figure 1: Examination Purposes 
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Issues 

The review of the literature revealed several issues that had an impact on which 
examination mode faculties chose for their comprehensive examinations In order to understand 
the relevance of each issue to the participants, we asked them to tell which mode was related 
most with each issue, least with each issue, and of the two remaining examination modes, which 
was related more with the issue. Results are summarized in Table I below 
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Issues 



Student Anxiety 
Public Acceptance 
Rigor 

Relevance to Program 

Relevance to Future of 
Candidate 

Faculty Work Load 
Potential for Dishonesty 



Table 1 
Most 



Oral 
(41.0%) 

Handwritten 
(41.0%) 

1 landuTitien/Alternative 
(23.1%) 

Allcmalivc 
(23.1%) 

Alternant c 
1 46.2%) 

Alternate e 
(53.8%) 

Allcmalivc 
(46.2%) 



Least 



Allcmalivc 
(64.1%) 

Alternative 
(64.1%) 

Alternaiive/No answer 
(30.8%) 

Allcmalivc 
(25 6%) 

I lanJunllen 
1 30 8%) 

Dial 
V 53 S%> 

Oral 
(35 9%) 



Respondents reported strong feelings about alternative assessments on many of the issues 
we asked about This method received the highest percentage of "most" ratings on four issues, 
and tied with handwriting on a fifth. Alternative assessments also received the highest percentage 
of "least" ratings on three issues, tying with "no answer" cn a fourth Alternative assessment 
methods were seen to be the most relevant of the four modes, both to the program, with 23 1% of 
the respondents considering them thus, and to the future of the degree candidate. *ilh 46 2% of 
the respondents giving it the "most" rating. However, alternative a^csssr^nts also seen to 
cause the greatest faculty w orkload (53 8% of respondents giving the;* « "most" rating) and to 
provide the greatest potential for dishonesty (46.2%). The alternative assessments tied with 
handwritten examinations as being the most rigorous. Alternative assessments were perceived to 
cause the least student anxiety (64. 1%), to be the least publicly acceptable (64 1%) and to be least 
relevant to the degree program (25.6%). They tied w ith "no answer" on the issue of rigor with 
30.8% considering it least rigorous or refusing to rate the modes on this issue 

Handwritten examinations were considered most publicly acceptable (41%). while oral 
examinations were perceived to cause the most student anxiety (4 1%) Interestingly, oral 
examinations were also seen to require the least faculty work (53 8% citing it as "least" in this 
issue) and the least potential for dishonesty (35 9%). I landwritten examinations were conquered 
least relevant to the future of the degree candidate (30 8%) 

Some unexpected responses bear explanation, as they had an impact on the results 
reported. For instance, 18% respondents stated that there was no dilVeience in administering a 
comprehensive examination where handwritten responses were required and administering a 
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comprehensive examination wheie responses were typed on the computer. The lack of difference, 
in their pciccption. made impossible discriminations among the examination modes on several 
issues In addition, 2 t% ol the departments surveyed allowed both modes of examination This 
finding reflects an important change from perceptions just a few years ago, when security issues 
and equipment malfunction issues made computer use a matter of question, rather than a matter of 
fact 

Several respondents (JO 8%) felt that the rigor of an examination was unrelated to mode 
That is, they felt that the issue of rigor related to the difficulty of the questions and to the 
expectations to which a student was held They saw these as unrelated to how the student was 
examined. We felt, on the other hand, that the mode of examination might determine the 
difficulty of questions or the rigor of expectation, and so decided to ask for distinctions. 

Alternative assessment modes, such as portfolios or performances, were associated most 
frequently with the most relevance, both to the program of study and to the future of the degree 
candidate Alternative assessment was also associated most frequently with the most faculty 
workload and the most potential for dishonesty This negative pair, perceived faculty workload 
and the perceived potential for dishonesty, may account for the relatively low percentage of 
respondents reporting that they use alternative assessment (25.6%~see Table 2), despite the 
perception of relevance. Alternative assessments were also perceived most often to be least 
publicly accepted and least rigorous, which again, may account for their relatively infrequent use 
at this point. We felt that the strong feelings regarding alternative assessments warranted further 
examination to determine who was reporting these strong feelings, and whether or not different 
groups reported different perceptions. 

Further examination through crosstabulation revealed that indeed, those who do not use 
alternative assessment modes are more likely to have reported negative attributes for this form of 
examination For instance, of those reporting that alternative assessment is least publicly 
accepted, 80% do not oiler alternative assessment modes. Indeed, 90 9% of those not offering 
alternative assessment report that it is least publicly accepted. While 0% of those using 
alternative assessment methods consider it least rigorous, 66.7% of those not offering alternative 
assessment consider it least rigorous of the examination modes. 60.9% of those not offering 
alternative assessment say that it has the most potential for dishonesty Both those who offer 
alternative assessment and those who do not seem to agree, though, that alternative assessment 
requires most faculty work of the assessment modes 70% of those offering alternative 
assessment consider it most work-intensive, as do 50% of those who do not ofTer alternative 
assessment 

Those who do not offer alternative assessment seem to have beliefs that match their 
policies. Crosstabulations regarding positive attributes showed that of those who do offer 
alternative assessment modes, 70% consider them most rigorous Many (55.6%) also considered 
alternative assessment to have the most program relevance 
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Discussion 

This preliminary re-examination of issues, purposes, and procedures for administering 
comprehensive examination resulted in several interesting findings. Most of the departments 
surveyed required a traditional comprehensive examination, taken over the course of several days 
and several hours. It is interesting, though, that while the literature indicated that there might be 
some question as to whether or not computers should be used to write comprehensive exams, 
91.2% of the departments using handwritten examinations also permitted students to use 
computers to write exams. This suggests that the use of computers is commonplace, and no 
longer an issue to most departments. The issue of anxiety while using computers, raised in some 
previous studies, seems not to be a consideration any more. 

What does seem to be an issue of concern for departments, at least in the area of 
educational administration, is the use of alternative assessments. One quarter of those responding 
said that they used alternative assessments, which is of some note. What is clearly noteworthy, 
though, is the strong feelings expressed regarding the use of alternative assessments. Alternative 
assessments received strongly positive or strongly negative ratings on all issues studied in this 
report. Heads of departments not offering alternative assessment modes tended to give them 
"most" ratings on negative issues, and "least" ratings on positive issues. The converse was true of 
departments offering alternative assessments. 

It may be that this concern surrounding alternative assessment reflects a changing purpose 
for administering comprehensive examinations While the emphasis on synthesis dating back to 
the thirteenth century remains, there is a growing trend to address the need to apply learned 
information. Alternative assessments are often viewed as ways for students to apply the 
information they have learned in classes in real ways. 

However, favoring application may necessarily lead to tension with those who expect to 
maintain the tradition of ensuring standard experiences and know ledge of graduates. That is, 
alternative assessments are often individualized, taking the form that best suits the student and his 
or her committee. This seems to preclude standardization in the traditional sense. All students 
would not have the same experiences, nor would they be held accountable for the same 
information. 

The fact that mean scores for examination purposes that imply valuing standardization are 
lower than those implying a focus on application and synthesis reflects a trend. The valuation of 
purposes implied by alternative assessment processes may lead more departments to consider, and 
perhaps use alternative assessment models to test their doctoral students Indeed the public push 
to expect graduate programs to provide relevant real world experiences to their students may also 
serve to push the comprehensive examination process toward new procedures, and away from 
those that have prevailed since the thirteenth century. 
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